Economical Inversion of Large Text Files

To provide keyword-based access to a large text file it is usually necessary to invert the file and create an inverted index that storeso for each word in the file, the paragraph or sentence numbers in which that word occurs. Inverting alarge file using traditional techniques may take as much temporary disk space as is occupied by the file itself, and consume a great deal of cpu time. Here we describe an alternative technique for inverting large text files that requires only a nominal amount of temporary disk storage, instead building the inverted index in compressed form in main memory. A program implementing this approach has created a paragraph level index of a I32 Mbyte collection of legal documents using 13 Mbyte of main memory; 500 Kbyte of temporary disk storage; and approximately 45 cpu-minutes on a Sun SPARCstation 2. @ Computing Systems, Vol. 5 . No. 2 ' Spring 1992 125